11L INT6 XSA-all + EMA + VE — ttt_bpb 1.1487#1216
11L INT6 XSA-all + EMA + VE — ttt_bpb 1.1487#1216SoHarshh wants to merge 9 commits intoopenai:mainfrom
Conversation
train_gpt_v2.py: - LZMA compression support (COMPRESS=lzma env var) - Full Hessian GPTQ: gptq_quantize_weight() + collect_gptq_hessians() (GPTQ_ENABLED=1 activates post-training column-wise quantization) train_gpt_v3.py (Parallel Muon): - All replicated-param all_reduces now launched async simultaneously so NCCL can pipeline them (saves ~3-5ms/step vs serial blocking calls) - Removed redundant .contiguous() in non-XSA attention path run.sh: - v11_proxy: 1-GPU smoke test for 11L INT6 stack - v11_int6_xsaall: 11L INT6 + XSA-all + LZMA + VE (train_gpt_v2.py) - v11_gptq: same + GPTQ_ENABLED=1 (train_gpt_v2.py) - v11_banked: 11L INT6 + XSA-all + LZMA + VE + Parallel Muon (train_gpt_v3.py)
Community Review — 11L INT6 XSA-all + EMA + VE — ttt_bpb 1.1487BPB: 1.1487 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern) What I found in the code (head SHA The TTT path at line 1017 implements the score-first-per-chunk pattern: each chunk is scored under Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=10, vocab=1024, code=78732 B, SMOKE_TEST_PASS Verdict: LOOKS CLEAN. Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass. Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually. Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=10, vocab=1024, code=78732 B, SMOKE_TEST_PASS. Classification via deterministic AST-based |
12L Banked + Parallel Muon + Value Embeddings — ttt_bpb 1.1571
Best submittable result: ttt_bpb = 1.1571, 16.47MB (seed 1)
Architecture
Also included: 11L INT6 XSA-all experiment (quality record, unsubmittable)